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ABSTRACT 

The recent fervor and reemergence of research in 
neural networks has its reasons. The most important are 
the ability of these systems to store vast numbers of 
complex patterns, the ability to recall these patterns in 
0(1) time (i.e., speed of pattern retrieval is independent 
of number of stored patterns), and the ability to recall 
these stored patterns using fuzzy or incomplete cues. 

In this paper, a brief history of the field will be 
reviewed and some simple concepts will be described. 

In addition, some neural network based avionics re- 
search and development programs will be reviewed. 

The concluding remarks will stress the need for the 
United States Air Force (USAF) and the National 
Aeronautics and Space Administration (NASA) to as- 
sume a leadership role in supporting this technology. 


INTRODUCTION 

The System Evaluation Branch of the Avionics 
Laboratory at Wright-Patterson Air Force Base is cur- 
rently working under a charter to transfer learning re- 
search to exploratory development of intelligent 
electronic combat systems. Neural networks have been 
identified by this group as having great potential for 
solving a variety of difficult problems encountered in 
military avionics. 

The purpose of this paper is to show the need for new 
approaches in developing intelligent systems for the 
USAF. It can be argued that this need also applies to 
NASA and the aerospace industry in general. The ar- 
gument for why neural networks have the potential for 
satisfying this need will be given by introducing some 
important properties of neural networks. A brief his- 
torical perspective of the field and the current trends in 
the technology will also be provided. In addition, a 
brief description of the research and development 
programs being conducted and planned by the Software 
Development Group will be given. 

NEED FOR NEW APPROACHES IN THE AIR FORCE 
TO DEVELOPING INTELLIGENT SYSTEMS 

The environments in which our military aircraft and 
weapons systems must operate in have become increas- 
ingly complex and hostile with the advancement of 
technology. Survival will depend on developing 


autonomous, flexible avionics systems that can adapt 
and learn from a highly dynamic and hostile 
environment. However, this is a tremendous challenge 
due to the complexity of these systems and their 
environments. The usual problem domains encountered 
in electronic combat systems, for example, can be 
characterized as follows: A usually small number of 
resources must be managed and allocated to satisfy mul- 
tiple constraints and optimization criteria. These sys- 
tems are capable of multiple responses under multiple 
threat and/or target environments. Changes to the en- 
vironment usually occur very rapidly and sometimes 
unexpectedly. These systems must process a tremendous 
amount of information under conditions of novelty, 
deception, incomplete data, and noise. A further crucial 
requirement is that these processes must be accomplish- 
able in real-time. 

Artificial Intelligence (AI) is one approach to develop- 
ing "intelligent” systems, but current AI technology has 
many limitations. The problem domains under which 
most AI technology has been developed are very dif- 
ferent than the problem domains of many military and 
aerospace applications. The problem domains most ex- 
pert systems have dealt with have been quite narrow, 
ideal, and free of noise. Most importantly, processing 
time has not been a critical factor. AI and other tradi- 
tional problem solving techniques have had difficulty 
dealing with many areas such as machine vision, 
automatic target recognition, situation assessment, and 
resource planning and control, to name a few. The real- 
time constraints have been one of the factors contribut- 
ing to the difficulty in developing AI based solutions to 
the problems mentioned above. 

O’Reilly & Cromarty (1985) have formally defined real- 
time system performance as the requirement that a 
system’s response to environmental stimuli occur in 
provably finite time (i.e., 0(1) time response). The 
authors show that current AI and traditional problem 
solving approaches cannot prove this time response and 
go on to say: 

"...our analysis indicates that there is no reason to 
expect conventional system design approaches 
from either school to yield effective, provable 
real-time performance.” 

They further propose that parallelism is one way of 
achieving this performance. 

This analysis is consistent with the general acceptance 
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in the A I community for the need of parallelism in 
their problem solving approaches. 

There has been considerable work in recent years in 
parallel processing, but developments in hardware have 
far outstripped the programmers ability to effectively 
use these systems. We are having problems developing 
parallel algorithms. This problem is exemplified by the 
title of a recent paper. Programming for parallelism: 
The state of the art of parallel programming and what a 
sorry state that art is in (Karp, 1987). 

Because of the limitations and slow progress in current 
AI research and development, especially as it relates to 
real-world military and aerospace operations, there has 
been a growing need to re-evaluate research strategies. 
One alternative approach which has a strong potential 
for satisfying the Air Force’s need for intelligent 
systems, is neural networks . 


NEURAL NETWORK SYSTEMS: THEIR 
PROPERITIES 

Neural networks have properties which seem to offer 
solutions to many of the difficult problems encountered 
in machine learning, vision, speech, pattern recognition, 
and real-time resource planning and control. These 
properties are all interrelated, making it difficult iden- 
tify the most important one. The remainder of this sec- 
tion will concentrate on explaining these properties. 

Most neural networks are modeled after or resemble 
some of the structure and function of biological brains 
and nerve cells (neurons), thus their name. These sys- 
tems are composed of interconnected processing ele- 
ments (PEs) or "neurons" which process information in 
parallel. The PEs have multiple inputs (from the output 
of other neurons or from external stimuli) and a single 
output. This output may in turn branch out to yet other 
PEs or the outside world. Neural networks are in- 
herently parallel processing systems. 

An important class of neural networks have the ability 
to learn and adapt in response to environmental 
changes. In these neural networks, the PE’s have self 
adjusting weights associated with their input channels 
(i.e., the conductance of the interconnections change 
with experience). This self adjusting of network 
parameters is the basis of learning in neural networks 
and is one of the most important characteristics of these 
systems. 

One very useful way of interpreting the dynamics of a 
neural network is as an energy field undergoing changes 
over time. One can think of this energy field as a flat 
sheet (it is actually a multidimensional surface). As the 
network interacts with its environment, wells or basins 


Although there is still considerable disagreement 
among psychologists on the principles of information 
processing of the brain, and many neurological func- 
tions and cellular mechanisms have not been resolved, 
mathematical models of certain structures and functions 
of the brain have been developed with characteristics 
similar to known neurological functions. 


are created or formed on this flat sheet over time. If 
the job of the network is to identify or categorize sig- 
nals of some kind, these wells represent the learned 
categories. The input stimuli can be thought of as 
marbles. As new marbles (input stimuli) fall onto this 
contoured sheet (energy field), the marbles will roll into 
the closest basin. Marbles that fall into a particular 
well are similar to the marbles that created the well to 
begin with. This brings us to the next set of related 
properties of neural networks. These systems are 
capable of associating arbitrary input states with the 
nearest energy basin (identification, category, or 
response). In addition, these systems decide what the 
appropriate features of the input states are in order to 
make the classifications or responses. Therefore, neural 
networks can act as associative memories, nearest 
neighbor pattern classifiers, and feature detectors 
(Kohonen, 1984; Kosko, 1986 and 1987a). 

A very important result in neural network research, the 
Cohen/Grossberg Theorem (Cohen & Grossberg 1983), 
was popularized in a similar finding by Hopfield (1982 
and 1984). This theorem states that the energy of a 
class of neural networks, called Crossbar Associative 
Networks (CANs), converges to a finite set of equi- 
librium points. The energy of the system is defined as a 
global Liapunov function and the equilibrium points 
are the local minima of that function. Not only is con- 
vergence guaranteed, but the time required to converge 
to those equilibrium points does not depend on the 
number of those points. In other words, CANs respond 
in 0(1) time. This is a characteristic of every neural 
network. 

Just as in conventional AI programs, knowledge repre- 
sentation is of utmost importance in neural networks. 
But knowledge is distributed throughout a massively in- 
terconnected processor architecture. For example, a 
certain neural network might have the concept of an 
airplane represented in its network. That concept will 
be distributed among many PEs and each PE will con- 
tain small pieces of information about other concepts; 
maybe tank, helicopter, jeep, etc. Due to the networks 
ability to utilize distributed knowledge representations 
which are supported by massive numbers of parallel 
elements, these networks are fault tolerant. Neural 
networks have been shown to exhibit graceful degrada- 
tion of performance as more PEs become inoperative 
(Anderson, 1983). One can understand why this occurs 
from the example of the airplane above. If one or two 
elements which contain information about that airplane 
are damaged, the remainder of the network may contain 
enough of the concept "airplane" to use that information 
effectively in some type of process. If any piece of 
hardware or software in conventional computers be- 
comes corrupted, there will be system failure. 


Many networks have been developed in which 
knowledge was not distributed. Each PE represented 
one and only one concept. In these experiments other 
properties and capabilities of neural networks were 
being examined which did not require distributed 
representations. 
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One final, very important characteristic which is sure to 
have a considerable impact on the aerospace industry, is 
that these systems process information without the use 
of computer programs. What is required is the 
specification and development of an architecture of in- 
terconnected PEs for a given problem. Each PE of the 
neural network is governed by a system of 
mathematical equations which can be implemented 
directly in electronic circuitry. After an architecture is 
defined, the neural network is then put through a train- 
ing or learning stage. It is in this stage that the system 
learns the appropriate I/O mappings with either the 
help of a "teacher” or "critic", or on its ow^ if enough a 
priori information is built into the system. Still 
other systems can learn continuously as they interact 
with their environments. 

Before leaving this section, a brief, high level descrip- 
tion of the mathematical equations governing a neural 
network will be given. The typical PE is governed by 
usually two or three differential equations (or dif- 
ference equations when dealing in discrete time). One 
equation determines the activity or state of the PE, 
another determines the change in conductances (or the 
final values of the conductances after the network 
settles to a stable state) of the PE’s input channels, and 
the third equation determines the output of the PE. 
When the PEs are governed by two equations, the ac- 
tivity of the PE is usually incorporated into the output 
equation. The activity equations are usually some func- 
tion of the sum of the weighted inputs. The output 
equations are usually a nonlinear function of the ac- 
tivity (either sigmoid or linear threshold). And the 
change in input conductances are usually some function 
of the inputs, output, and the conductances themselves. 

These dynamical equations come in a variety of forms 
which have either evolved or have been added over the 
years to give us a large and rich repertoire today. This 
variety reflects the diverse and interdisciplinary back- 
ground of the researchers in the field: neuroscience, 
psychology, physics, mathematics, engineering, and 
computer science. 

A number of important attributes of neural networks 
have been discussed. It must be emphasized, however, 
that the engineering process of developing architectures, 
especially for real world problems, is still in its infancy. 
Convergence theorems for many classes of important 
neural networks have not been found. Fortunately, we 
do have enough empirical data to suggest that conver- 
gence proofs to some of these systems may be found. 
Other problems include strict limits on the amount of 
data storage imposed on a neural network of given size 
and the ability of associative memories to create 
spurious energy minima (Kosko 1987a). The important 
point to stress is that neural networks offer a tremen- 
dous potential for solving many difficult problems 
which solutions have not been previously, or acceptably 
found. But before this potential is realized, much work 
needs to be done. 


More detailed discussions on the different classes 
of learning and how these are accomplished in neural 
networks are discussed in Duda & Hart (1973), Barto & 
Sutton (1981), and Barto (1985). 


HISTORY AND CURRENT TRENDS 

In this section, a brief history and the current trends in 
neural network research will be introduced in order to 
give a general feel for the field. For a comprehensive 
review see Levine (1983). Barto (1984) also presents a 
more in depth review than the one found here. Prob- 
ably the best introduction to neural networks is 
provided by Rumelhart, et al (1986). This work also in- 
cludes research more appealing to those with AI and 
cognitive science backgrounds. 

The early concepts of processing information by a net- 
work of simple linear threshold elements were intro- 
duced by McCulloch and Pitts (1943). They developed 
very simple linear threshold processing elements with 
boolean output which were interconnected via positive 
and negative input lines. Their results generated much 
excitement for they showed any logical function could 
be performed by some configuration of such networks. 

The next major milestone was achieved by Hebb (1949) 
when he postulated a mechanism for long term memory. 
This mechanism required a structural change to the 
neuron: 

"When the axon of cell A is near enough to 
excite a cell B and repeatedly or persist- 
ently takes part in firing it, some growth 
process or metabolic change takes place in 
one or both cells such that A’s efficiency, 
as one of the cells firing B, is increased 
(Hebb, 1949, p. 64)" 

This hypothesis was later interpreted mathematically as 

d_ wj - xjy 

dt 

where w- is the weight associated with the ith input to 
the neuron, Xj is the ith input signal from another 
neuron, and y is the output of the neuron in question. 

This rule has had a tremendous impact on neural net- 
work research for it has been used in one form or 
another in virtually every learning neural network 
conceived. 

It wasn’t until the late 50s and early 60s that neural 
networks were developed along the lines in which we 
are familiar with today. McCulloch and Pitts’ ideas of 
interconnected linear threshold elements and Hebb’s 
ideas of long term memory were integrated into very 
useful devices. Two such systems deserve special 
attention: The Perceptron, developed by Rosenblatt 
(1962) and the Adaline (for adaptive logic element), 
developed by Widrow (1962). Both of these systems 
were similar in that they were based on a single adap- 
tive layer of neurons and on an error correcting 
mechanism. The difference between the desired 
response and the actual response was fed back to the 
adaptive layer through a series of training trials until 
the network converged to a solution in provable finite 
time. 

Unfortunately for Rosenblatt and for neural network 
research for the next 20 years or so, Rosenblatt made 
claims which seemed unfounded to several of his 
contemporaries. This led to Minsky and Papert’s (1969) 
critical response to the Perceptron (see Rumelhart, et al, 
1986, pp. 151-159). Minsky and Papert showed a number 
of limitations of single layered adaptive networks and 
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also raised the issue of the credit assignment problem in 
multilayered networks of error correcting elements. 

The credit assignment problem arises as a result of the 
inability of cells within the interior layers of the net- 
work to know what fraction of the total error they are 
responsible for. These problems have been solved in a 
variety of ways since then (Parker, 1982 and 1985, and 
Rumelhart, et al, 1985), but in those days they raised 
alarming questions. 

Minsky and Papert’s book was devastating to Rosenblatt 
and neural network research. The book was a sign to 
many that research should be directed towards symbolic 
processing and heuristics. This approach is what we 
know as AI today. The push for this approach was also 
being heavily influenced by the growing field of cogni- 
tive psychology (for a historical view from this perspec- 
tive see Gardner, 1985). At this same time, behaviorist 
psychology was in decline. This also helped sway re- 
search funds away from neural networks since the 
issues involved in neural networks were highly reminis- 
cent of the issues the behaviorists were grappling with: 
stimulus/response chains, reinforcement, and behavior 
based on microstructural concepts. 

Widrow, on the other hand, was extremely successful 
applying his Adaline and Madaline (for many Adalines) 
to signal processing. His adaptive signal processing 
techniques have been applied to system modeling (i.e., 
imitating system behavior), inverse system modeling, 
adaptive control systems, adaptive interference 
canceling, and to adaptive antenna arrays (Widrow, 
1985). It is also interesting to point out Widrow’s 
achievements in the 60s. His Knobby Adaline (a 
hardware implementation of his adjustable threshold 
element, Widrow 1962) was able to recognize patterns 
regardless of noise (10%), rotation (90 degrees), left and 
right translation, and size (25%). The Avionics 
Laboratory, at Wright-Patterson Air Force Base owns a 
film of a pole balancing experiment performed by 
Widrow. A small cart with a pole attached to the top of 
the cart by a pivot was placed on a short track. The 
Madaline was able to keep the pole balanced by control- 
ling the cart’s movement after a series of training trials. 
In that same film, Widrow’s students are shown training 
a Madaline to translate spoken words in three languages 
to type written English. One may wonder whether the 
neural network "nuclear winter" that ensued would have 
taken place if these results would have been marketed 
as vigorously as the limitations to the technology at the 
time. 

Although neural network modeling fell from grace after 
Minsky and Papert’s book, very important work con- 
tinued throughout the 70s. Fukushima (1975) and von 
der Marlsburg (1973) developed systems based on the 
visual structures of biological brains. Kohonen (1972) 
and Wilshaw, et al (1970 and 1971) were early pioneers 
in the area of associative memories. Amari, et al (1974 
and 1977) made an important contribution through his 
research in associative memories and their relation to 
thermodynamics. Klopf (1972 and 1979) introduces the 
concept of the neuron as a goal oriented or goal seeking 
agent (heterostat). Rescorla and Wagner (1972) 
developed a model which exhibited a variety animal 
learning phenomena. 


•The USAF supported Widrow’s research as well as 
other neural network research in those early days. 


The most prolific contributor to this field has been 
Stephen Grossberg, of the Center for Adaptive Studies, 
at Boston University. Grossberg has addressed all the 
main issues in neural networks from 1967 (Grossberg, 
1967) to date in approximately 130 papers and 4 books. 
He has approached his research with rigorous mathe- 
matics and has proved some of the most important 
theorems in the field. He has investigated and written 
about memory, animal learning behavior, cognition, 
speech, language, vision, and motor control. He’s col- 
lected his most important work in three volumes 
(Grossberg, 1982, 1987a, and 1987b). 

John Hopfield of Caltech and Bell Labs is, perhaps, the 
one most responsible for reigniting the field. In two ar- 
ticles (Hopfield, 1982 and 1984), Hopfield, expanding 
on previous work on crossbar associative networks 
(CANs), made connections between CANs and Ising spin 
glass models of ferromagnetism. Hopfield’s papers 
made a strong impact on the physics and optics com- 
munity in a series of conference presentation. Hopfield 
and Tank (1985) further publicized the information 
processing capabilities of neural networks by develop- 
ing a CAN system which had the ability to find near 
optimum solutions to a traveling salesman problem. In 
other words, they developed an 0(1) time approximate 
solution to a NP-hard problem using neural networks 
(see Hecht-Nielsen, 1986 for this discussion). Interest in 
the field has mushroomed in academia, the Department 
of Defense, and throughout the aerospace industry since 
Hopfield’s 1982 paper. 

Today, theoretical work continues at a fast pace from 
many of the original pioneers mentioned above and 
from scores of others entering this exciting field. Over 
200 papers were presented in 16 sessions at the Institute 
of Electrical and Electronics Engineers (IEEE) spon- 
sored First Annual International Conference on Neural 
Networks in San Diego, California, between the 21st 
and 24th of June, 1987. Nearly 2,000 people were at- 
tracted to this event. The 80s have also brought much 
needed work in the hardware implementation of neural 
networks. In the past, almost all work was simulated on 
general purpose computers. Experiments could run for 
days in those early years. Special purpose processors are 
coming to market today which can significantly in- 
crease processing speed. For many applications, these 
are sufficient for real-time processing. For more dif- 
ficult problems such as vision or target recognition, 
much larger networks will be required. If these net- 
works are to be flown in spacecraft and aircraft, they’ll 
have to be implemented in silicon, optics, or a combina- 
tion of both. Fortunately, work is well under way ad- 
dressing this need. The following is only a small sam- 
pling of optical and electronic neural network research: 
Cruz- Young & Tam, 1985; Dunning, et al, 1986; Graf & 
deVegvar, 1987a and 1987b; Fisher, et al, 1986; Psaltis 
and Abu-Mostafa, 1985; Psaltis and Farhat, 1985; 
Sivilotti, 1985; and Soffer, 1986. 


SYSTEM EVALUATION BRANCH’S (AAAF) R&D EF- 
FORTS 

Basic research in neural networks has matured to a 
point suitable for translation into exploratory 
development. AAAF’s efforts are aimed at advancing 
neural network research in both the signal processing 
and cognitive processing areas for avionics applications. 
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The ultimate goal is to merge both areas of research and 
develop the technology for providing intelligent 
avionics sensor systems for the USAF. We are specifi- 
cally addressing the avionics domain from the level of 
sensors and emitters in electronic combat applications. 

This research is part of a long term program. Intelligent 
Avionics , which is in general addressing the issue of 
making avionics adaptive. Both contracted and in- 
house efforts will be conducted under this program. 

Two contracts are currently being managed in the 
neural network area. The first is the Adaptive Network 
Cognitive Processor (ANCP), a one year effort which was 
awarded to TRW in San Diego, California. The purpose 
of this program is to develop a prototype system which 
builds an inner model of its environment in the form of 
cognitive maps and uses this model for reasoning, 
planning, or problem solving. The exact problem 
domain is a high level situation assessment and response 
system for pilot aiding. This is a H proof of concept” 
program. A TRW Mark III neurocomputer is being used 
for neural network design and simulation and will be 
delivered as part of the prototype. 

The second program, the Adaptive Network Sensor 
Processor , will apply neural network 
associative memory and pattern recognition technology 
to a military radar warning system for providing 
identification, categorization, and classification of pre- 
viously experienced and novel radar signals in a noisy 
and corruptive environment. A comparison between 
this new approach to radar signal identification and 
conventional means of signal processing will be ac- 
complished before system delivery. There are two con- 
tractors working on this program: Booz-Allen & 

Hamilton, Inc. from Arlington, Virginia and Texas In- 
struments from Dallas, Texas. A Hecht-Nielsen 
Neurocomputer Company (HNC) ANZA neurocomputer 
will also be used by Booz-Allen & Hamilton. Texas In- 
struments will be using their array processor board, the 
Odyssey, in conjunction with their Explorer work sta- 
tion for developing and simulating the neural network 
and environment. 

Follow-on efforts for both of these programs are being 
planned. The Adaptive Network Avionics Resource 
Manager (ANARM) will apply what is learned in ANCP 
to a specific electronic combat system. The Adaptive 
Network Radar Signal Processor will integrate ANSP 
with a response module to provide closed loop learning. 
Hardware implementation issues will also be 
investigated. These programs are scheduled to start in 
fiscal year 1988 and fiscal year 1989 respectively. 

In-house research is also being conducted under a 
program entitled Real-time Adaptive Avionics . As part of 
this effort, a neural network design tool was developed 
and implemented on an LMI (now Giga Mos) Lambda 
LISP Machine. The Artificial Neural Design Environment 
(ANDE) has been used to investigate the application of 
KlopFs (1986) Drive-Reinforcement Neuronal Model to 
a simulated avionics control problem. The ultimate goal 
of this research is to transfer a neural network architec- 
ture to electronic combat groups which can perform 
real-time, adaptive resource management and control. 
Support for this research is being pursued from the Air 
Force Office of Scientific Research (AFOSR). 


CONCLUSION 

The United States Air Force and the National 
Aeronautics and Space Administration should assume a 
leadership role in advancing neural network research 
and development efforts because of the tremendous 
potential for providing adaptive, fault-tolerant 
aerospace systems. We have available to us a viable 
technological alternative which offers potential solu- 
tions to such complex problems as data fusion, machine 
vision, automatic target recognition, resource planning 
and control, and adaptive system control. The impor- 
tant characteristics of neural network, which are sum- 
marized below, must be exploited and used in innova- 
tive ways in order for this potential to be realized. 
Neural networks are parallel processing systems that can 
respond in 0(1) time. These systems can learn and 
adapt to their environment and are fault tolerant to 
damage. And finally, neural networks can process in- 
formation without the need of computer programs. The 
foreseen software explosion and crisis could be 
diminished or alleviated. 

Neural network technology will not supplant current 
computer science and software development where these 
are more appropriate. Rather, hybrid systems are envi- 
sioned with each technology performing what it does 
best. New developments in neural network technology, 
however, have the potential to revolutionize and greatly 
enhance intelligent information processing for our 
country’s defense and space science. It is also clear that 
the USAF and NASA should steer the research efforts 
in this area in order that neural network technology 
develops in a manner suited to aerospace requirements. 
The System Evaluation Branch of the Avionics 
Laboratory at Wright-Patterson Air Force Base is com- 
mitted to develop and exploit this "new” technology for 
developing intelligent avionics systems. 
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